<!DOCTYPE html><!--Author: Pranav Rajpurkar 2016-->
<html>

<head>
    <meta charset="utf-8">
    <title>Multimodal Analogical Reasoning over Knowledge Graphs</title>
    <meta name="description"
        content="Multimodal analogical reasoning over knowledge graphs is a new task which requires multimodal reasoning ability with the help of background knowledge. To support it, we construct a Multimodal Analogical Reasoning dataSet (MARS) and a multimodal knowledge graph MarKG.">
    <meta http-equiv="X-UA-Compatible" content="IE=edge,chrome=1">
    <meta name="viewport" content="width=device-width, initial-scale=1, maximum-scale=1, user-scalable=no">
    <meta property="og:image" content="/logo.png">
    <link rel="image_src" type="image/png" href="./logo.png">
    <link rel="shortcut icon" href="./favicon.ico" type="image/x-icon">
    <link rel="icon" href="./favicon.ico" type="image/x-icon">
    <link rel="stylesheet" href="./bower_components/bootstrap/dist/css/bootstrap.min.css">
    <link rel="stylesheet" href="./stylesheets/layout.css">
    <link rel="stylesheet" href="./stylesheets/index.css">
    <script async defer src="https://buttons.github.io/buttons.js"></script>
    <script src="./javascripts/analytics.js"></script>
</head>

<body>
    <div class="navbar navbar-default navbar-fixed-top" id="topNavbar" role="navigation">
        <div class="container clearfix" id="navContainer">
            <div class="rightNav">
                <div class="collapseDiv"><button class="navbar-toggle collapsed" type="button" data-toggle="collapse"
                        data-target="#navbar" aria-expanded="false" aria-controls="navbar"><span
                            class="glyphicon glyphicon-menu-hamburger"></span></button></div>
                <div class="collapse navbar-collapse" id="navbar">
                    <ul class="nav navbar-nav navbar-right">
                        <li><a href="./introduction.html">INTRODUCTION</a></li>
                        <li><a href="https://zjunlp.github.io/project/MKG_Analogy/">LEADERBOARD</a></li>
                        <li><a href="https://github.com/zjunlp/MKG_Analogy">GITHUB</a></li>
                        <li><a href="https://huggingface.co/spaces/zjunlp/MKG_Analogy">DEMO</a></li>
                    </ul>
                </div>
            </div>
            <div class="leftNav">
                <div class="brandDiv"><a class="navbar-brand" href="./">MKG Analogy</a></div>
            </div>
        </div>
    </div>
    <div class="cover" id="topCover">
        <div class="container">
            <div class="row">
                <div class="col-md-12">
                    <h1 id="appTitle">MKG Analogy</b></h1>
                    <h2 id="appSubtitle">Multimodal Analogical Reasoning over Knowledge Graphs</h2>
                </div>
            </div>
        </div>
    </div>
    <div class="cover" id="contentCover">
        <div class="container">
            <div class="row">
                <div class="col-md-5">
                    <div class="infoCard">
                        <div class="infoBody">
                            <div class="infoHeadline">
                                <h2>About MarKG</h2>
                            </div>
                            <p>Analogical reasoning is fundamental to human cognition and holds an important place in various fields.
                                Thus, we introduce the multimodal analogical reasoning task, which can be formulate as link prediction 
                                without explicitly providing relations. To support it, we collect an Multimodal Analogical Reasoning 
                                dataSet (MARS) and a multimodal knowledge graph (MarKG). Among them, MARS has 10,685 training, 1,228 
                                validation and 1,415 test instances. MarKG contains 11,292 entities, 192 relations and 76,424 images, 
                                include 2,063 analogy entities and 27 analogy relations.</p>
                                <br /><p>For more tails about the task and the datasets, please refer to our ICLR 2023 paper:<br /></p><a class="btn actionBtn"
                                href="http://arxiv.org/abs/2210.00312"> ZHANG, LI, CHEN ET AL.</a>
                            <div class="infoHeadline">
                                <h2>Getting Started</h2>
                            </div>
                            <div class="infoHeadline">
                                <h3>MarKG and MARS</h3>
                            </div>
                            <p> MarKG and MARS are distributed under <a
                                    href="http://creativecommons.org/licenses/by-sa/4.0/legalcode">CC BY-SA 4.0 license</a>
                                , download the textual data of them by following links:
                            <ul class="list-unstyled">
                                <li><a class="btn actionBtn inverseBtn" href="https://drive.google.com/file/d/1ZfgTKve4sYW-Bww14Af3AuZ-U0PTvgZ2/view?usp=sharing"
                                        download>MarKG (1MB)</a></li>
                                <li><a class="btn actionBtn inverseBtn" href="https://drive.google.com/file/d/1EL9---lH8t3AltjCwgKpApXMho4VCZdz/view?usp=sharing"
                                        download>MARS (10.9MB)</a></li>
                            </ul>
                            <div class="infoHeadline">
                                <h3>Image data</h3>
                            </div>
                            <p>The image data (18.72G) of MARS and MarKG can be be downloaded through the
                                <a href="https://drive.google.com/file/d/1AqnyrA05vKngfEbhw1mxY5qEoaqiKsC1/view?usp=share_link">Google Drive</a>
                                or the <a href="https://pan.baidu.com/s/1WZvpnTe8m0m-976xRrH90g">Baidu Yun(TeraBox)(code:7hoc)</a> .</p>
                            </p>
                            <div class="infoHeadline">
                                <h3>Checkpoints</h3>
                            </div>
                            <p>We provide the best checkpoints of our transformer-based models during the fine-tuning and pre-training phrases at this
                                <a href="https://drive.google.com/drive/folders/1ul9vC93t_e5t_fDj3zzgJKqPoPn_jgsX?usp=share_link">link</a>.</p>
                            </p>
                            <p>For more introduction and guidence, please check out our <a href="https://github.com/zjunlp/MKG_Analogy">Github repository</a>.</p>
                            
                            <!-- <div class="infoHeadline">
                                <h2>About the Leaderboard</h2>
                            </div>
                            <p>To facilitate diversified research about named entities, we release all the data (including the test set) of the three tasks. We encourage the community to do research beyond these settings (such as open/ unsupervised/ continual NER or entity typing/ linking, etc). Enjoy!
                                But we still maintain a leaderboard to record the peer-reviewd results. </p> -->

                            <div class="infoHeadline">
                                <h2>Connection</h2>
                            </div>
                            <p> If you have any question about the multimodal analogical reasoning task and the datasets
                                MARS and MarKG, or you want to submit your results to update the leaderboard, feel free
                                to emtail to us:</p>
                            <br>
                            <ul>
                                <li style="color: rgb(226, 163, 152);">
                                    <a href="mailto:zhangningyu@zju.edu.cn" tyle="font-size: 0.95rem; color:darkred;">
                                        zhangningyu@zju.edu.cn;
                                    </a>
                                </li>
                                <li style="color: rgb(226, 163, 152);">
                                    <a href="mailto:leili21@zju.edu.cn" tyle="font-size: 0.95rem; color:darkred;">
                                        leili21@zju.edu.cn;
                                    </a>
                                </li>
                                <li style="color: rgb(226, 163, 152);">
                                    <a href="mailto:xiang_chen@zju.edu.cn" tyle="font-size: 0.95rem; color:darkred;">
                                        xiang_chen@zju.edu.cn;
                                    </a>
                                </li>
                            </ul>
                            <br>
                            <p>If you use or extend our work, please cite the paper as follows:</p>
                            <br>
                            <pre style="font-size: 1.0rem; background-color: rgb(245, 238, 238);">@inproceedings{zhang2023multimodal,
    title={Multimodal Analogical Reasoning over Knowledge Graphs},
    author={Ningyu Zhang and Lei Li and Xiang Chen and Xiaozhuan Liang and Shumin Deng and Huajun Chen},
    booktitle={The Eleventh International Conference on Learning Representations },
    year={2023},
    url={https://openreview.net/forum?id=NRHajbzg8y0P}
}</pre>
                        </div>
                    </div>
                </div>
                <div class="col-md-7">
                    <div class="infoCard">
                        <div class="infoBody">
                            <div class="infoHeadline">
                                <h2>Leaderboard</h2>
                            </div>
                            <table class="table performanceTable">
                                <tr>
                                    <th>Rank</th>
                                    <th>Model</th>
                                    <th>Hit@1</th>
                                    <th>Hit@3</th>
                                    <th>Hit@10</th>
                                    <th>MRR</th>
                                </tr>
                                <tr>
                                    <td>
                                        <p>1</p><span class="date label label-default">Oct 01, 2022</span>
                                    </td>
                                    <td style="word-break:break-word;">
                                        <p class="institution">MKGformer (MART)</p>
                                        <a class="link"
                                            href="https://dl.acm.org/doi/10.1145/3477495.3531992"> (SIGIR '22)
                                        </a>
                                        
                                    </td>
                                    <td><b>0.301</b></td>
                                    <td><b>0.367</b></td>
                                    <td><b>0.408</b></td>
                                    <td><b>0.341</b></td>
                                </tr>
                                <tr>
                                    <td>
                                        <p>2</p><span class="date label label-default">Oct 01, 2022</span>
                                    </td>
                                    <td style="word-break:break-word;">
                                        <p class="institution">ViLBERT (MART)</p>
                                        <a class="link"
                                            href="https://proceedings.neurips.cc/paper/2019/hash/c74d97b01eae257e44aa9d5bade97baf-Abstract.html"> (NeurIPS '19)
                                        </a>
                                        
                                    </td>
                                    <td><b>0.256</b></td>
                                    <td><b>0.312</b></td>
                                    <td><b>0.347</b></td>
                                    <td><b>0.292</b></td>
                                </tr>
                                <tr>
                                    <td>
                                        <p>3</p><span class="date label label-default">Oct 01, 2022</span>
                                    </td>
                                    <td style="word-break:break-word;">
                                        <p class="institution">FLAVA (MART)</p>
                                        <a class="link"
                                            href="https://arxiv.org/abs/2112.04482"> (CVPR '22)
                                        </a>
                                        
                                    </td>
                                    <td><b>0.264</b></td>
                                    <td><b>0.303</b></td>
                                    <td><b>0.319</b></td>
                                    <td><b>0.288</b></td>
                                </tr>
                                <tr>
                                    <td>
                                        <p>4</p><span class="date label label-default">Oct 01, 2022</span>
                                    </td>
                                    <td style="word-break:break-word;">
                                        <p class="institution">RSME (ANALOGY)</p>
                                        <a class="link"
                                            href="https://dl.acm.org/doi/10.1145/3474085.3475470"> (ACM MM '21)
                                        </a>
                                        
                                    </td>
                                    <td><b>0.266</b></td>
                                    <td><b>0.298</b></td>
                                    <td><b>0.311</b></td>
                                    <td><b>0.285</b></td>
                                </tr>
                                <tr>
                                    <td>
                                        <p>5</p><span class="date label label-default">Oct 01, 2022</span>
                                    </td>
                                    <td style="word-break:break-word;">
                                        <p class="institution">VisualBERT (MART)</p>
                                        <a class="link"
                                            href="https://arxiv.org/abs/1908.03557"> (arxiv '19)
                                        </a>
                                        
                                    </td>
                                    <td><b>0.261</b></td>
                                    <td><b>0.292</b></td>
                                    <td><b>0.321</b></td>
                                    <td><b>0.284</b></td>
                                </tr>
                                <tr>
                                    <td>
                                        <p>6</p><span class="date label label-default">Oct 01, 2022</span>
                                    </td>
                                    <td style="word-break:break-word;">
                                        <p class="institution">IKRL (ANALOGY)</p>
                                        <a class="link"
                                            href="https://doi.org/10.24963/ijcai.2017/438"> (IJCAI '17)
                                        </a>
                                        
                                    </td>
                                    <td><b>0.266</b></td>
                                    <td><b>0.294</b></td>
                                    <td><b>0.310</b></td>
                                    <td><b>0.283</b></td>
                                </tr>
                                <tr>
                                    <td>
                                        <p>7</p><span class="date label label-default">Oct 01, 2022</span>
                                    </td>
                                    <td style="word-break:break-word;">
                                        <p class="institution">TransAE (ANALOGY)</p>
                                        <a class="link"
                                            href="https://ieeexplore.ieee.org/document/8852079/"> (IJCNN '19')
                                        </a>
                                        
                                    </td>
                                    <td><b>0.261</b></td>
                                    <td><b>0.285</b></td>
                                    <td><b>0.293</b></td>
                                    <td><b>0.276</b></td>
                                </tr>
                                <tr>
                                    <td>
                                        <p>8</p><span class="date label label-default">Oct 01, 2022</span>
                                    </td>
                                    <td style="word-break:break-word;">
                                        <p class="institution">ViLT (MART)</p>
                                        <a class="link"
                                            href="https://arxiv.org/abs/2102.03334"> (ICML '21)
                                        </a>
                                        
                                    </td>
                                    <td><b>0.245</b></td>
                                    <td><b>0.275</b></td>
                                    <td><b>0.303</b></td>
                                    <td><b>0.266</b></td>
                                </tr>
                            </table>
                        </div>
                    </div>
                </div>
            </div>
        </div>
    </div>
    <nav class="navbar navbar-default navbar-static-bottom footer">
        <div class="container clearfix">
            <div class="rightNav">
                <div>
                    <ul class="nav navbar-nav navbar-right">
                        <li><a href="./">MKG Analogy</a></li>
                        <li><a href="https://github.com/zjunlp">ZJUNLP</a></li>
                    </ul>
                </div>
            </div>
        </div>
    </nav>
    <script src="./bower_components/jquery/dist/jquery.min.js"></script>
    <script src="./bower_components/bootstrap/dist/js/bootstrap.min.js"></script>
</body>

</html>
